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The Growing State Data Capacity 


Ever since the 2001 passage of the No Child Left Behind Act (NCLB), states have faced 
a federal mandate to improve their ability to collect and analyze education data for the 
purpose of tracking and reporting the progress of their education reform efforts. With the 
assistance of federal funds, virtually all of the states are developing comprehensive state 
data systems that track students’ academic progress longitudinally throughout their 
primary, secondary, and even post-secondary careers. The Data Quality Campaign, an 
independent national organization, has played a complementary role in supporting states’ 
efforts to improve their student data systems. 

[See sidebar.] Beyond student assessment, 
the comprehensive data systems developed 
by most states also enable them to correlate 
the academic gains of their K-12 students 
with their assignment to particular teachers 
and thus to assess the effectiveness of 
individual teachers as well. 


The role of the federal government has gone beyond simply requiring states to increase 
their data collection and reporting capacity, however. Under the Statewide Longitudinal 
Data Systems Grant Program, the U.S. Department of Education has contributed more 
than half a billion dollars to the states to better understand what policies, investment 
strategies, and school intervention efforts are working or not working, and why. In other 
words, the push is on for state education agencies to do more than monitor compliance 
and report out data; increasingly, states are expected to strengthen their ability to do 
research on the programs and policies they implement. 

Beyond reporting out education data and making it meaningful and actionable, it is also 
important for states to be able to use their ever-more sophisticated data to analyze the 
impact of the various policies and programs they have implemented in an effort to 
improve student and school performance. Indeed, accountability-conscious policy-makers 
and the public have increasingly come to insist that education policies and programs be 
evidence-based - a demand that is directly related to the increased amount of data 
presumed to be available to provide the evidence expected. Although states appear to 
have become increasingly adept at using the data at their disposal for administrative 
purposes, it is far from clear that many states successfully use their data to undertake the 
kind of research and analysis that evidence-based policy development requires. 
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The U.S. Department of Education 
provides substantial funding - $500 million 
through FY201 1 - to state educational 
agencies to design, develop, and 
implement statewide P-20 longitudinal 
data systems to capture student data from 
preschool to high school, college, and 
career. These data systems can support 
decision-making and continuous school 
improvement at all grade levels. 


States’ Data Analysis Gap 


Several commentators who have tracked the implementation of NCLB indicate that 
although state officials are complying with the monitoring and reporting requirements of 
the act, they are unprepared for the new challenges of developing and implementing 
policies and strategies that are evidence-based. In a 2006 paper, 1 Gail Sunderman and 
Gary Orfield write that “Requiring states to intervene and force change in schools and 
districts requires a very different sort of capacity and expertise than that required for 
monitoring or tunneling funds to local districts.” Similarly, an Education Sector column 
online 2 noted that “despite states’ and districts’ tremendous progress in building data 
systems, policymakers are not yet routinely using these new data to improve 
accountability systems, support performance management processes, evaluate programs, 
or influence resource allocation decisions.” If this trend continues, then similar reports on 
the reauthorized Elementary and Secondary Education Act (ESEA) are likely to find that 
most states and local education agencies are maintaining massive comprehensive 
longitudinal education data systems but that their efforts at data-driven decision-making 
rarely involve sufficient analyses. 

This inability of states to make the most effective use of the growing amount of education 
data they collect is not due to any lack of appreciation for the potential of data and 
research to inform or to the lack of competence of state personnel. Rather, it is an issue 
of inadequate state research and analysis capacity. The unprecedented - and increasing 
- amount of education data that states are now required or encouraged to collect 
imposes a huge human resources demand on states. Simply maintaining comprehensive 
statewide educational data systems, let alone doing research that uses their data, 
requires a significant number of highly skilled staff, including individuals who are expert in 
analyzing and interpreting complex data releases. 

It is easy to blame inadequate federal funding and tight state budgets for the insufficient 
research and analytic capacity in so many state education agencies. But while more 
funding undoubtedly would be helpful, it is likely that part of the analytic capacity problem 
can be attributed to administrative priorities that end up sustaining outdated 
organizational structures. Many state education agencies are concerned, first and 
foremost, with demonstrating compliance with recent state reform mandates. In meeting 
this priority, they seem to have focused on building school and district capacity at the 
expense of investing in research. This is an unfortunate irony; in an era when building 
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capacity largely requires the choosing and/or modifying policies and programs based on 
evidence, a commitment to research must be regarded as central. 

State education officials are aware of the importance of such a commitment. A 2007 
Center on Education Policy survey conducted by Angela Minnici and Deanna Hill 3 
reported a common view among state officials that insufficient numbers of staff, lack of in- 
house expertise, and inadequate federal and state funding were the major impediments 
to implementing the requirement to provide technical assistance to schools in need of 
improvement. In particular, the survey found that “close to half of the states interviewed 
emphasized the need for federal technical assistance (and often funding) to help states 
and school districts develop the capacity to collect and analyze the complex assessment 
and other data required by NCLB.” A 2008 Government Accountability Office (GAO) 
survey 4 corroborated the CEP survey finding, reporting that officials in the vast majority of 
states believe that they could benefit from more technical assistance and analytic support, 
such as more information on promising improvement practices and “what works.” And 
another 2008 report to education leaders by CEP 5 noted, “In terms of central capacity 
issues, states are telling us ... we don’t have sufficient numbers of staffing and we don’t 
have the right staff to do the job that we need to do... States not only lack the proper staff 
to assist at-risk schools, but they also lack the resources to properly assess the 
effectiveness of the policies when they are in practice, as well as the effectiveness of 
strategies and programs that are aimed at improving performance.” 

The technical support that state education agencies require to carry out such 
assessments goes well beyond the kind of off-the-shelf help that is readily available. 

States need customized information and sustained assistance to address their specific 
priorities, economic condition, political culture, and state educational agency capacity. 
Direct and ready access to highly-qualified researchers and scientifically-trained analysts 
will be critical if state agency heads are expected to overcome the gap between states’ 
aspirations - and the public’s expectations - for data-driven analysis of policies and 
programs and the limitations of their present capacity. 

To meet the near-term challenges associated with “doing more with less,” state and local 
policy-makers will need more and better analysis of the data so far collected. However, 
making evidence-based decisions often requires sophisticated use of statistical 
techniques, policy simulations, and cost-benefit analyses, as well as the “art” involved in 
the recognition that the data in state data systems are neither perfect nor comprehensive. 
Thus, for example, the analysis needed might involve the development of metrics to 
monitor the effects of increased class size or to indicate how the performance of 
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individual schools and their students can influence teacher workforce decisions. Seriously 
tackling these questions will require the science of estimating statistical models of the 
progress schools are making relative to the changing federal educational performance 
standards. But it also will require the art of translating the outputs of models into policies 
and helping education policy-makers make informed judgments about tolerable risks and 
tradeoffs - including costs - in continuing, altering, or curtailing programs. Unless states 
have this sort of multi-dimensional research and analytic capacity, they simply will not be 
able to take full advantage of the data systems they are building and will be unlikely to 
develop and implement the policies and programs that will address their education needs 
most effectively. 
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Compromising Reforms 


This analysis gap threatens to compromise the success of a wide range of current 
education reform efforts. There are a number of current initiatives, for example, to 
develop and implement innovative teacher evaluation and teacher compensation systems. 
These include the federal Race to the Top (RTT) program, the Gates Foundation’s 
Partnership for Effective Teaching, and other independent state or district programs. 

Most of these initiatives involve an assessment of teachers’ effectiveness on the basis of 
their students’ performance on state achievement tests and other measures of teachers’ 
skill. For this to be possible, states require not only adequate data but also the resolution 
of a number of highly technical and important issues such as the following: 

• Which model of value-added assessment is the most reliable and appropriate for 
the states’ student assessment and data systems? 

• Are the states’ student achievement tests, including scoring standards, 
sufficiently rigorous and sensitive to reflect true student abilities and learning 
gains, even at the very low and high ends of the scoring spectrum? 

• What additional measures can be validly employed in assessing teachers’ 
effectiveness, and how should the different measures (including student 
achievement test performance) be weighted to produce a combined assessment? 

• What important patterns do the results of the teacher assessments show, e.g., 
about the characteristics of the teachers who are evaluated as most effective or 
about the success of various teacher preparation programs in producing capable 
teachers? 

Although many state education officials are aware of the need to address such 
considerations, the reports previously cited suggest that most states have not addressed 
them as deeply and concretely as they should. 

Another result of a state’s limited capacity to research and analyze its data might be a 
misreading of state education conditions or a misdiagnosis of underlying problems. An 
effort to embark on a new and ambitious teacher recruitment program, for example, might 
be based on teacher attrition data that paint a much bleaker picture than is, in fact, the 
case. This would be the result, for example, of a failure to distinguish between teachers 
who transfer from one district to another and teachers who move out of state or leave the 
profession - not an uncommon data limitation. The result would likely be an over- 
production of teachers and a waste of funding (e.g., on teacher recruitment incentives). 
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Similarly, low student achievement scores in eighth grade mathematics may be attributed 
to poor teacher preparation programs when the deeper, unrecognized problem is that a 
state’s requirements for K-8 licensure - which often guide preparation program 
requirements and curricula - are insufficient to ensure that eighth grade teachers’ 
mathematics knowledge is adequate. Or, on the other hand, a state in fact may have 
increased the mathematics course requirements for K-8 teacher licensure, but with no 
appreciable impact on students’ performance on the state mathematics assessment 
because the deeper problem is that the academic standards at many of the universities 
that prepare the state’s teachers are too low to ensure that graduates have an adequate 
grasp of the subject even with additional coursework. 

Finally, inadequate analysis and research related to an important state education reform 
effort can result not only in the failure of the effort to reach its full potential but in 
unforeseen harmful consequences. A classic example is California’s decision to reduce 
K-3 class sizes in 1996, with the result that the overall quality of the teacher workforce 
declined even as education expenditures increased, and student achievement failed to 
rise as was anticipated. In the current push to equalize the distribution of effective 
teachers among the states’ schools, there are certainly possibilities for negative 
consequences if the effort is undertaken without an adequate analysis. Particularly in 
subjects with shortages of teachers (such as science and mathematics), it may be 
impossible to have enough effective teachers to go around, and attempts to achieve 
equal distribution by lowering the overall average teacher effectiveness of some schools 
to raise it in another could simply reduce student achievement in the schools from which 
effective teachers were drawn away. In addition, such a scenario might have the further 
consequence of alienating the students and their parents at the adversely-affected 
schools, undermining trust and confidence in the district, and leading parents to transfer 
their children to private schools or even move out of the district. 
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Expanding States’ Research Capacity 


Much is at stake in ensuring that states have the capacity they need to meet the 
demands and opportunities for research and analysis that their powerful data systems 
provide. This capacity must be sustained over time so that states can continue to 
maintain the quality of their data, monitor their education climate and the long-term 
impacts of policies and programs, and remain well-informed about the promise and 
limitations of various education strategies they might be interested in adopting or scaling 
up. Such a state research and analysis agenda can almost never be sustained in-house; 
in the best of economic times, few state officials - even those in state education agency 
research departments - are called upon to engage in rigorous research. Even as states 
modernize their state education agencies, they are likely to require assistance from 
external resources in meeting the level of research and analysis that will serve their 
needs. 

In some states, their major research universities may have faculty and staff capacity that 
could help states address their research-related needs in education. Other states may 
have access to smaller nonprofit organizations that engage in state-specific education 
research. The research agendas of the major universities and the other nonprofits may 
not completely match the priority research needs of state education agencies, however. 
And, in any case, states may not have sufficient funds to engage external researchers 
even if they are available to address state education research priorities. 

Federally funded research and technical assistance organizations, and particularly the 
Regional Education Laboratories (RELs), can be expected to play an important role in 
facilitating states’ ability to meet their needs for research and analysis. The RELs make 
available the services of expert researchers whose mission is specifically to focus, on 
request, on identified state and local needs for assistance with research and researched- 
based analysis. Being external to any public or private state agency, these researchers 
are not invested in specific outcomes or points of view and are not beholden to particular 
state stakeholders. Thus, they can provide a completely independent analysis for the 
states they serve. 

Our own organization, CNA, houses one of the regional educational research laboratories, 
REL Appalachia, which serves the states of Kentucky, Tennessee, Virginia, and West 
Virginia. CNA embeds a highly-trained field scientist in the state education agency of 
each of these states for the purpose of assisting state policy-makers and program 
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managers to more effectively analyze, disaggregate, and use data on the performance of 
schools and students in the state. The field scientists assist their assigned states in 
improving their education data systems and expanding them to incorporate workforce 
and post-secondary education information. Providing an independent scientific research 
analyst to the states has proved to be very successful. The embedded researchers have 
been able to develop a solid understanding of each state’s culture and its education and 
political picture and also to gain the trust and respect of state officials to whose technical 
assistance needs the researchers can readily respond. 

The ability of the RELs to serve in such a capacity could be further enhanced by a 
modification of their mission to incorporate a greater responsibility for conducting 
independent analyses of state and local educational performance data and for helping 
states with their “everyday” needs for evidenced-based decision-making. Currently, the 
RELs engage in research on three different levels: 

1 . Multi-year random controlled trials at the most rigorous extreme of the research 
spectrum 

2. “Fast Response Studies” in the middle of the spectrum, which last for up to a 
year, are rigorously peer reviewed, and allow a variety of research methods to be 
employed 

3. Shorter-term, research-based briefs that provide much more immediate 
responses to specific requests from state officials, either in the form of reviews of 
relevant research literature or targeted analyses of state data. 

All three of these kinds of assistance are valuable, but there needs to be much more 
support and facilitation of the RELs’ role in providing less formal, shorter-term assistance 
that can respond to immediate state needs for analysis . 

Regardless of how states attempt to address the growing need for research and analysis 
to meet federal mandates for collecting and using sophisticated data, the states will 
require additional resources to be able to do so. Continuing federal incentives will serve 
to strengthen the capacity of states’ data systems, and the upcoming reauthorization of 
the Elementary and Secondary Education Act (ESEA) is an opportunity to boost the 
payoff of those data systems. Ensuring that states have greater resources to interpret 
and analyze the increasingly complex integrated data systems should be a priority in 
ESEA reauthorization, and that priority should be manifest not only in new dollars for 
state-level research and analysis but also in incentives to carry out the specific kinds of 
research that are of greatest state benefit. 
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A number of options to accomplish this can be considered within the framework of the 
ESEA reauthorization. State education agencies could be given a new discretionary set- 
aside authority, for example, that allows a state agency to devote a very small 
percentage of its total state allocation - for ESEA, Individuals with Disabilities Education 
Act (IDEA), and Carl D. Perkins Vocational and Technical Education Act - to employ 
independent researchers and analysts. No matter where the dollars originate, however, 
they should be appropriated specifically for activities designed to increase the ability of 
state agencies to make and use evidence-based decisions. Congress could require the 
oversight needed to ensure that these funds are spent on appropriate activities (e.g., for 
new analytical studies and not to supplement current administrative activities), while still 
allowing states the flexibility they need to address their ongoing and emerging decision- 
making needs. 

Federal officials responsible for authorizing the programs directed to improve elementary 
and secondary education need to acknowledge that, without addressing states’ needs for 
expanded research capacity, not much of the anticipated $40 billion annual appropriation 
will result in “evidence-based” decisions. Few state and local educational administrators 
have the time or skills to determine the effectiveness and impact of programs based on a 
systematic analysis and interpretation of school and student performance data. Although 
surveys suggest that state and local school district officials are doing an adequate job in 
responding to the requirement for collecting and accumulating educational assessment 
data, the present prospect for making productive use of these data is less encouraging. 

The possibility for improving that prospect rests with those responsible for crafting and 
updating federal education laws. If they can incorporate new provisions that allocate a 
portion of a state’s set-aside for evaluation and technical assistance - especially by a 
cadre of independent researchers and analysts - they can greatly increase the likelihood 
that states’ school improvement and intervention initiatives will be solidly evidence-based 
and rest on solid scientific ground. To be sure, providing states with such additional 
resources to close the analysis gap will not overcome all of the challenges associated 
with school improvement. It could, however, help states improve their ability to choose 
policies and school intervention investment strategies that are supported by independent 
research and analysis. 

Several years after the enactment of the Elementary and Secondary Education Act in 
1965, a Harvard University researcher found that the massive infusion of federal 
education dollars had little effect on strengthening state departments of education. He 
noted that the federal funding initiatives did not stimulate a rethinking of priorities or 
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promote a thorough overhaul of activities and failed to promote basic institutional 
change. 6 The Harvard report and more recent studies suggest that it is time to recognize 
that some states may not have the research and technical expertise readily available to 
guide the adoption and implementation of research and evidence-based school reform. 
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